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Introduction 


The  goal  of  this  project  is  to  develop  an  Evidence-based  Clinical  Decision  Support  System  (CDSS-EBM)  available  at  the 
point  of  care  which  will  improve  prognostication  of  life  expectancy  of  terminally  ill  patients  and  facilitate  the  hospice 
referral  process.  In  addition,  the  CDSS-EBM  will  be  expanded  with  a  pain  management  module  to  assist  physicians 
managing  terminally  ill  patients  or  patients  with  chronic  pain. 

Work  done  so  far: 

We  have  completed  the  research  and  development  (R&D)  committee  review  process  at  the  James  A.  Haley  veteran's 
hospital  and  have  obtained  the  approval  letter.  We  have  obtained  the  University  of  South  Florida's  (USF)  and  Tampa 
General  Hospital  (TGH)  institutional  review  board  (IRB)  authorization. 

We  have  submitted  the  required  paperwork  to  the  James  A.  Haley  Veteran's  hospital  to  obtain  the  authorization  from 
the  security  officer  and  the  final  approval  to  initiate  the  study  at  the  James  A.  Haley  Veteran's  hospital.  We  have 
submitted  the  required  documents  including  the  research  protocol  and  informed  consent  forms  to  the  regulatory 
specialist  with  the  Telemedicine  and  Advanced  Technology  Research  Center  (TATRC),  to  secure  the  approval  from  the 
USAMRMC  Office  of  Research  Protections  (ORP)  Human  Research  Protections  Office  (HRPO).  It  should  be  noted  that  all 
our  work  so  far  has  been  theoretical  using  publicly  available  data-set  (SUPPORT)  as  we  are  still  awaiting  for  the  final 
approval  to  begin  our  study  by  Human  Research  Protection  Office  (HRPO),  Office  of  Research  Protections  (ORP)  U.S. 
Army  Medical  Research  &  Materiel  Command  (USAMRMC).  Nevertheless,  we  believe  we  have  been  highly  productive 
and  able  to  accomplish  a  number  of  important  tasks. 

We  have  also  completed  the  hiring  of  personnel  for  the  first  retrospective  phase  of  this  project. 

Our  progress  regarding  each  of  the  tasks  outlined  in  the  Statement  of  Work  is  as  follows: 

Task  1 :  we  have  developed  the  case  report  form  that  will  allow  us  to  retrieve  data  from  the  Lifepath  Hospice's  charts. 
Data  to  be  retrieved  include  all  variables  used  in  the  SUPPORT  [3]  prognostication  model. 

Task  2:  We  have  acquired  the  original,  retrospective  dataset  used  to  develop  the  SUPPORT  prognostication  model.  We 
utilized  this  dataset  in  multiple  ways.  First  we  used  the  entire  dataset  to  compute  the  decision  curves  related  to  the 
SUPPORT  model  and  the  alternatives  of  referring  all,  or  no  patients  to  hospice.  The  decision  curves  are  used  in  decision 
curve  analysis  (DCA)  [4]  where  each  alternative  strategy  is  compared  based  on  the  decision  maker's  personal 
preferences  as  they  are  described  by  a  threshold  probability.  We  found  that  the  SUPPORT  prognostication  model  is  the 
preferable  decision  strategy  for  a  large  range  of  threshold  probabilities  (8%  to  91%). 

In  addition,  we  used  a  portion  of  the  SUPPORT  dataset  (patients  with  lung  cancer  only)  to  estimate  their  life  expectancy. 
For  this  task,  we  utilized  the  DEALE  [5]  prognostication  model  and  the  lung  cancer  evidence  tables  created  by  our  group 
(see  Task  4).  Since  the  SUPPORT  prognostication  model  is  reporting  probability  of  survival  and  not  life  expectancy,  we 
have  modified  the  SUPPORT  outcomes  in  order  to  compare  them  with  the  results  of  the  DEALE  model.  We  have 
concluded  that,  regarding  lung  cancer,  more  investigation  is  needed  to  formally  compare  the  two  models.  We  hope  that 
with  capturing  additional  data  from  the  retrospective  phase  this  will  be  possible,  once  we  get  approval  to  begin  data 
collection. 

Finally,  the  third  use  of  the  SUPPORT  dataset  was  to  develop  an  initial  version  of  a  Rough  Set  Theory  (RST)  [6] 
prognostication  model  for  life  expectancy.  This  version  of  the  RST  model  uses  the  majority  of  the  variables  included  in 
the  SUPPORT  model  and  it  has  shown  promising  prediction  results.  Further  investigation  is  required  to  determine  the 
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impact  of  reducing  the  number  of  variables  utilized  or  the  addition  of  qualitative  variables  such  as  physicians'  choices 
such  as  continuation  of  treatment  or  referral  to  hospice.  This  will  be  possible  when  prospective  phase  of  the  study  is 
approved  /commenced. 

Along  with  the  work  outlined  in  Task  2,  we  have  developed  a  novel  methodology  for  comparison  of  decision  strategies 
based  on  regret  theory  and  decision  curve  analysis.  Our  method  takes  into  consideration  personal  preferences  of 
decision  makers  by  quantifying  the  consequences  of  wrong  decisions.  In  fact,  we  have  developed  a  new  method  for 
elicitation  of  patients'  preferences,  which  represent  a  significant  achievement  toward  individualized  evidence-based 
decision-making.  A  paper  has  been  submitted  to  BMC  Medical  Informatics  and  Decision  Making  [1]  (revisions  stage)  and 
a  presentation  is  scheduled  for  the  Society  for  Medical  Decision  Making  Annual  meeting  [2],  (see  Appendix  for  full 
report). 

Task  3:  We  have  developed  a  pilot  web  based  version  of  the  CDSS-EBM  to  compute  life  expectancy  for  terminally  ill 
patients  based  on  the  DEALE  and  SUPPORT  prognostication  models.  This  CDSS-EBM  will  be  utilized  to  suggest  optimal 
strategies  for  particular  simulated  patients.  This  pilot  CDSS-EBM  will  facilitate  the  hospice  referral  procedure  in  three 
steps.  First,  the  probabilities  of  survival  and  death  for  a  specific  patient  are  computed  based  on  the  SUPPORT 
prognostication  model.  Then,  based  on  interviews  with  either  the  patient,  his/her  family,  or  his/her  physician  the  system 
elicits  personal  preferences  regarding  continuation  of  life-sustaining  vs.  palliative  care.  Finally,  using  regret  DCA,  the 
optimal  decision  for  the  specific  patient  is  suggested. 

Task  4:  We  are  in  the  process  of  conducting  a  systematic  review  addressing  the  prognosis  of  metastatic  lung  cancer 
patients.  We  have  finished  the  literature  search  and  found  39  Randomized  Controlled  Trials  (RCTs).  We  have  extracted 
data  for  the  outcomes  of  overall  survival,  type  of  treatment  and  patient  characteristics  etc.  from  these  RCTs.  We  have 
calculated  the  pooled  overall  median  survival  of  lung  cancer  patients  from  these  RCTs.  Similarly,  we  have  calculated  the 
pooled  proportion  of  mortality  among  patients  receiving  no  active  treatment  from  the  studies.  Finally  we  are  creating 
evidence  tables  for  Lung  cancer  disease  to  use  as  inputs  for  the  DEALE  prognostication  methodology. 

Task  5:  We  have  developed  a  pilot  Chronic  Pain  Management  Module  to  complement  the  CDSS-EBM.  Our  objective  is  to 
develop  a  reliable  dosage  conversion  system  as  well  as  a  knowledge  based  for  each  available  pain  medication.  To  that 
end,  we  have  extracted  data  on  dosage,  cycles,  route,  duration,  and  time  to  peak  effect  for  all  available  pain 
medications.  In  addition  to  the  dosage  conversion  system,  we  have  developed  evidence-based  web-based  expert  system 
for  pain  management  based  on  the  National  Comprehensive  Cancer  Network  guidelines.  The  system  is  currently  going 
through  the  final  programming  phase  and  will  be  tested  internally  and  in  the  clinic  once  we  get  the  approval  to  begin 
clinical  part  of  the  study. 

Key  research  accomplishments 

•  Development  of  case  report  for  data  extraction  from  Tampa  Bay  Lifepath  Hospice  Center's  data. 

•  Developed  decision  curves  for  the  SUPPORT  prognostication  model. 

•  Developed  the  first  version  of  the  Rough  Set  Theory  prognostication  model. 

•  Created  a  novel  methodology  that  combines  Decision  Curve  Analysis  and  Regret  theory  to  compare  multiple 
decision  strategies  based  on  the  decision  maker's  personal  attitudes  towards  each  strategy. 

•  Developed  a  new  method  for  elicitation  of  patient  preferences 

•  Developed  a  pilot,  web-based  version  of  the  CDSS-EBM  for  hospice  referral. 

•  Created  evidence  tables  for  Lung  cancer. 

•  Developed  a  pilot,  web-based  version  of  the  chronic  pain  management  module. 
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Reportable  outcomes 

1.  Journal  publication:  (appendix  1) 

A.  Tsalatsanis,  I.  Hozo,  A.  Vickers,  B.  Djulbegovic.  A  regret  theory  approach  to  decision  curve  analysis:  A  novel  proposal 
for  elicitation  of  decision  maker's  preferences.  Submitted  to  BMC  Medical  Informatics  and  Decision  Making;  Revisions 
stage, 

2.  Abstract  presentation:  (appendix  2) 

I.  Hozo,  A.  Tsalatsanis,  A.  Vickers,  B.  Djulbegovic.  A  regret  theory  approach  to  decision  curve  analysis.  To  be  presented  at 
SMDM  annual  meeting  2010. 

Conclusion 

We  have  already  begun  working  on  the  majority  of  tasks  described  in  the  Statement  of  Work.  We  believe  that  we  have 
closely  followed  the  grant's  timeline  where  we  could  control  the  work  process.  At  this  point,  we  are  still  working  on  fine- 
tuning  theoretical  aspects  of  our  work,  while  waiting  for  the  final  approval  of  clinical  part  of  the  study.  Once  we  obtain 
the  approval  from  the  HRPO  office,  we  can  start  collecting  retrospective  data  from  Tampa  Bay  Life  path  hospice  center 
and  evaluate  the  performance  of  our  prognostication  models.  Our  research  findings  so  far  can  be  summarized  as 
follows: 


•  Based  on  theoretical  model,  we  found  that  deciding  based  on  the  SUPPORT  prognostication  model  is  the 
preferable  strategy  for  a  large  range  of  threshold  probabilities  (8%  to  91%)  when  compared  to  the  strategies  of 
referring  all  patients  to  hospice  or  referring  no  patients  to  hospice. 

•  We  concluded  that,  regarding  lung  cancer,  more  investigation  is  needed  to  formally  compare  the  predictive 
performances  of  SUPPORT  and  DEALE  models. 

•  Based  on  the  area  under  the  curve  measure,  the  RST  model  performs  better  than  SUPPORT  in  patient 
classification.  However,  further  investigation  is  required  to  determine  the  impact  of  reducing  the  number  of 
variables  utilized  or  the  addition  of  qualitative  variables  such  as  physicians'  values  regarding  hospice  referral. 

In  the  near  future  we  intend  to  investigate  further  details  regarding  the  development  of  the  RST  prognostication 
methodology  and  its  application  to  both  SUPPORT  and  hospice  datasets.  We  are  expecting  a  journal  publication  as  a 
result  of  this  effort.  Our  goal  is  develop  the  appropriate  theoretical  framework  that  will  facilitate  the  hospice  referral 
process  based  on  outcomes  of  multiple  prognostication  models.  Our  plan  is  to  develop  evidence-based  decision-support 
system  that  will  help  both  better  referral  to  hospice  as  well  help  with  pain  management.  Ultimately,  the  usefulness  of 
our  system  will  depend  how  well  it  performs  when  tested  in  clinical  setting. 

Next  Steps 

Our  immediate  next  step  is  to  complete  a  retrospective  phase  of  the  study.  Based  on  the  results  of  the  retrospective 
phase  of  the  study,  we  will  decide  on  the  final  site  for  the  prospective  phase  of  the  study:  Tampa  General  Hospital  (TGH) 
or  James  A.  Haley  Veteran's  hospital.  TGH  refers  more  patients  to  hospice  and  logistically  would  allow  us  faster 
completion  of  the  study.  In  addition,  if  the  retrospective  phase  shows  that  the  gender  is  important  for  accurate 
prediction  and  the  subsequent  decision-making,  we  will  then  likely  conduct  study  at  TGH  to  preserve  the  generalizibility 
of  the  results.  Nevertheless,  the  final  decision  will  be  made  after  the  retrospective  phase  of  the  study  is  completed. 
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Abstract 


Background 

Decision  curve  analysis  (DCA)  has  been  proposed  as  an  alternative  method  for  evaluation  of  diagnostic  tests,  prediction 
models,  and  molecular  markers.  However,  DCA  is  based  on  expected  utility  theory,  which  has  been  routinely  violated  by 
decision  makers.  Decision-making  is  governed  by  intuition  (system  1),  and  analytical,  deliberative  process  (system  2), 
thus,  rational  decision-making  should  reflect  both  formal  principles  of  rationality  and  intuition  about  good  decisions.  We 
use  the  cognitive  emotion  of  regret  to  serve  as  a  link  between  systems  1  and  2  and  to  reformulate  DCA. 

Methods 

First,  we  analysed  a  classic  decision  tree  describing  three  decision  alternatives:  treat,  do  not  treat,  and  treat  or  no  treat 
based  on  a  predictive  model.  We  then  computed  the  expected  regret  for  each  of  these  alternatives  as  the  difference 
between  the  utility  of  the  action  taken  and  the  utility  of  the  action  that,  in  retrospect,  should  have  been  taken.  For  any 
pair  of  strategies,  we  measure  the  difference  in  net  expected  regret.  Finally,  we  employ  the  concept  of  acceptable  regret 
to  identify  the  circumstances  under  which  a  potentially  wrong  strategy  is  tolerable  to  a  decision-maker. 

Results 

We  developed  a  novel  dual  visual  analog  scale  to  describe  the  relationship  between  regret  associated  with  "omissions" 
(e.g.  failure  to  treat)  vs.  "commissions"  (e.g.  treating  unnecessary)  and  decision  maker's  preferences  as  expressed  in 
terms  of  threshold  probability.  We  then  proved  that  the  Net  Expected  Regret  Difference,  first  presented  in  this  paper,  is 
equivalent  to  net  benefits  as  described  in  the  original  DCA.  Based  on  the  concept  of  acceptable  regret  we  identified  the 
circumstances  under  which  a  decision  maker  tolerates  a  potentially  wrong  decision  and  expressed  it  in  terms  of 
probability  of  disease. 

Conclusions 

We  present  a  novel  method  for  eliciting  decision  maker's  preferences  and  an  alternative  derivation  of  DCA  based  on 
regret  theory.  Our  approach  may  be  intuitively  more  appealing  to  a  decision-maker,  particularly  in  those  clinical 
situations  when  the  best  management  option  is  the  one  associated  with  the  least  amount  of  regret  (e.g.  diagnosis  and 
treatment  of  advanced  cancer,  etc). 
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Background 


Decision  making  is  often  governed  by  uncertainty  that  inevitably  affects  the  overall  decision  process.  In  their 
efforts  to  model  uncertainty,  decision  theorists  have  proposed  many  methodologies  with  the  majority  of  them  having 
been  based  on  statistics  and  probability[l-4],  information  theory  and  entropy[5],or  possibilistic  approaches  such  as  fuzzy 
logic[6,  7], 

In  clinical  medical  research,  much  effort  has  been  invested  in  developing  decision  support  systems  for  diagnosis 
and  treatment  of  various  clinical  conditions  such  as  management  of  infectious  diseases  in  an  intensive  care  unit,  chronic 
prostatitis,  or  liver  surgery[8-12]to  name  a  few  examples.  Most  of  these  systems  are  based  on  probabilistic  prediction 
models.  Even  though  prediction  models  have  been  shown  to  be  generally  superior  and  potentially  complementary  to 
physicians'  prognostications  [13-15],  historically  they  have  not  fulfilled  decision  makers  expectations  to  help  improve 
decision-making.  One  reason  for  this  is  that  most  probabilistic  medical  decision  support  systems  are  based  on  expected 
utility  theory  that  humans  often  violate[14,  16,  17],  In  addition,  most  models  in  medicine  do  not  incorporate  decision¬ 
makers'  preferences,  which  in  addition  to  having  reliable  evidence,  is  the  key  to  rational  decision-making[18-20]. 

The  goal  of  this  paper  is  to  develop  a  novel  decision-making  approach  that  incorporates  the  decision  maker's 
attitudes  towards  multiple  treatment  strategies.  Our  goal  is  addressed  through  the  following  three  specific  aims.  First, 
we  deviate  from  the  traditional  expected  utility  theory  in  an  attempt  to  satisfy  both  formal  criteria  of  rationality  and 
human  intuition  about  good  decisions[18-22].  We  employ  regret  theory,  since  regret  is  a  cognitive  emotion  that 
combines  both  rationality  and  intuition,  which  are  key  elements  for  decision-making[22,  23],  to  develop  a  novel 
methodology  for  eliciting  decision  makers'  personal  preferences  .  Consequently  we  reformulate  decision  curve  analysis 
(DCA)[24,  25]  from  the  regret  theory  point  of  view  to  evaluate  alternative  treatment  strategies  and  to  integrate  both 
evidence  on  prognosis  and  treatment  with  the  decision  maker's  attitudes  and  preferences[26-28].  Finally,  we  identify 
circumstances  under  which  a  decision  maker  tolerates  a  wrong  decision. 

To  implement  our  approach,  we  first  compute  the  threshold  probability  at  which  the  decision  maker  is 
indifferent  between  alternative  actions,  based  on  the  level  of  regret  one  might  feel  when  he/she  makes  a  wrong 
decision.  We  then  employ  the  regret  based  DCA  to  identify  the  optimal  strategy  for  a  particular  decision  maker.  The 
optimal  strategy  is  the  one  that  brings  the  least  regret  in  the  case  that  it  is,  in  retrospect,  wrong.  We  also  show  how  to 
employ  a  prediction  model  to  estimate  the  probability  of  disease  for  a  patient  and  contrast  it  with  the  decision  maker's 
threshold  probability.  Finally,  we  incorporate  the  concept  of  acceptable  regret  in  the  decision  process  to  identify  the 
conditions  under  which  the  decision  maker  tolerates  a  potentially  wrong  decision. 

Methods 


Decision  analysis  based  on  regret  theory 

Figure  1  depicts  a  typical  decision  tree  describing  administration  of  treatment  guided  by  a  prediction  model. 
There  are  two  competing  strategies  (treat,  and  do  not  treat),  and  four  possible  outcomes  as  described  by  the 
combinations:  treat/  do  not  treat  and  necessary/unnecessary. 

In  Figure  1,  p  =  P{D  +)  is  the  probability  associated  with  the  presence  of  the  disease  as  estimated  by  a 
prediction  model;l  —  p  =  P(D  — )is  the  probability  associated  with  the  absence  of  the  disease,  and  Uu  i  e  [1,4],  are  the 
utilities  corresponding  to  each  outcome.  For  example,  U{\ s  the  utility  of  administering  treatment  to  a  patient  who  has 
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the  disease  (e.g.  treat  when  necessary),  and  U2 is  the  utility  of  administering  treatment  to  a  patient  who  does  not  have 
the  disease  (e.g.  administering  unnecessary  treatment).  Note  that  we  use  the  term  "treatment"  in  the  generic  sense  of 
health  care  intervention,  which  may  indicate  therapy,  procedure,  or  a  diagnostic  test. 

The  probabilistic  nature  of  prognostication  models  complicates  significantly  the  decision  process.  For  example,  if 
a  prediction  model  estimates  the  probability  of  a  patient  having  a  disease  equal  to  40%,  it  is  unclear  whether  this  patient 
should  receive  treatment  or  not.  A  solution  from  the  point  of  view  of  the  classical  decision  theory  is  to  employ  the 
concept  of  threshold  probability  Pt,  which  is  defined  as  the  probability  at  which  the  decision  maker  is  indifferent 
between  two  strategies  (e.g.  administer  treatment  or  not)[27,  29,  30],  Based  on  the  threshold  concept,  the  patient 
should  be  treated  if  p  >  Pt  and  should  not  be  treated  otherwise. 

However,  since  in  most  cases  decisions  are  made  under  uncertainty  and  can  never  be  100%  accurate  [23,  26,  28, 
31-34],  Thus,  after  a  decision  has  been  made  one  may  discover  that  another  alternative  would  have  been  preferable. 

This  knowledge  may  bring  a  sense  of  loss  or  regret  to  the  decision  maker[23,  26,  28,  31-34],  Regret  can  be  particularly 
strong  when  the  consequences  of  wrong  decisions  are  life  threatening  or  seriously  influence  the  quality  of  the  patient's 
life. 


Formally,  regret  can  be  expressed  as  the  difference  between  the  utility  of  the  outcome  of  the  action  taken  and 
the  utility  of  the  outcome  of  the  action  that,  in  retrospect,  should  have  been  taken  [23,  26,  28,  31-34],  Regret  can  be  felt 
by  any  party  involved  in  the  decision-making  process  (e.g.  patients  receiving  treatment,  patient's  proxies  or  physicians 
administering  treatment).  For  the  rest  of  this  paper  we  assume  that  the  decision  maker  is  the  treating  physician. 

We  first  employ  regret  theory  to  estimate  the  threshold  probability,  Pt ,  at  which  the  physician  is  indifferent 

between  alternative  management  strategies  (e.g.  administer  treatment  or  not).  In  order  to  accomplish  this,  we  describe 
regret  in  terms  of  the  errors  of  (1)  not  treating  the  patient  who  has  the  disease,  and  (2)  treating  the  patient  who  does 
not  have  the  disease. 


Figure  2  describes  the  derivation  of  regret  associated  with  each  strategy  based  on  the  utilities  of  each  action's 
outcome.  As  can  be  noted,  the  regret  associated  with  the  error  of  not  treating  the  patient  when  he/she  should  have 
received  treatment  (the  probability  of  disease  is  p>Pt),  Rg(Rx—,  D+),  is  equal  to  the  loss  in  benefits  of  treatment.  This 
can  be  expressed  as  the  difference  between  the  utility  of  receiving  treatment  and  having  the  disease,  and  the  utility  of 
not  receiving  treatment  and  having  the  disease  (U1-U3). 

Similarly,  the  regret  associated  with  treating  the  patient  who  should  not  have  received  treatment  (the 
probability  of  disease  is  p  <  Pt ),  Rg(Rx- F,  D— ),  is  equal  to  harms  incurred  due  to  treatment.  This  can  be  expressed  as 
the  difference  between  the  utilities  of  not  having  the  disease  and  not  receiving  treatment,  and  not  having  the  disease 
and  receiving  treatment  (U4-U2).  We  expect  no  regret  in  the  cases  of  correct  treat/no  treat  decisions,  Rg(Rx+,  D  +)  = 
Rg(Rx—,  D  — )  =  0.  The  difference  (U1-U3)  represents  the  consequences  of  not  administering  treatment  where 
indicated,  while  (U4-U2)  represents  the  consequences  of  administering  treatment  to  a  patient  who  does  not  need  it. 
Under  these  assumptions,  the  threshold  probability,  Pt  is  equal  to  [27,  29,  30]: 


Pt  = 


U  i  -U, 


1  + 


(1) 


u4-u2 

Equation  1  effectively  captures  the  preferences  of  the  decision  maker  towards  administering  or  not 
administering  treatment.  At  the  individual  level,  equation  1  shows  how  the  threshold  probability  relates  to  the  way  the 
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decision  maker  weighs  false  negative  (i.e.  failing  to  provide  necessary  treatment)  vs.  false  positive  (i.e.  administering 
unnecessary  treatment)  results[24,  25], 


Note  that  the  fraction  Ul  U 3  is  undefined  for  U4  —  U2  =  0,  which  means  that  in  this  situation  there  is  no  regret 

u4-u2 

associated  with  administering  unnecessary  treatment.  Under  these  circumstances,  Pt  =  100%,  indicating  that 
treatment  is  justified  only  in  case  of  absolute  certainty  of  disease  (p=100%),  a  realistically  unachievable  goal[26]. 

Elicitation  of  threshold  probability 

There  are  numerous  techniques  for  eliciting  the  decision  maker's  preferences  regarding  treatment 
administration  [35],  None  of  them  has  been  proven  to  be  better  than  the  other.  We  argue  that  any  attempt  to  measure 
people's  preferences  and  risk  attitudes  should  be  derived  from  an  underlying  theory  of  decision-making  that  can  be 
applied  to  a  problem  or  a  class  of  the  problems  at  hand.  We  approach  elicitation  of  preferences  by  capturing  people 
attitudes  (e.g.  physicians')  through  threshold  probabilities.  Normatively,  a  threshold  probability  reflects  indifference 
between  two  alternative  management  strategies. 

There  are  few  commonly  used  methods  to  assess  the  value  of  this  indifference  for  a  decision  maker  such  as  the 
standard  gamble,  and  the  time  trade-off  [35-37],  The  problem  is  that  both  standard  gamble  and  time  trade-off  are  time- 
consuming,  cognitively  more  complex  and  are  shown  that  can  lead  to  biased  estimates  of  people's  preferences  [36,  37], 
An  alternative  method  is  to  use  rating  scales,  such  as  visual  analog  scales  (VAS),  which  are  considerably  easier  to 
administer  and  better  understood  by  the  participants.  The  problem  with  analog  scales,  however,  is  that  they  cannot 
capture  health  states  trade-offs[36,  37], 

The  proposed  method  retains  the  simplicity  of  VAS  but  it  takes  into  account  the  consequences  of  possible 
mistakes  in  decision-making  by  utilizing  two  visual  analog  scales.  The  first  scale  aims  to  assess  the  regret  associated  with 
potential  error  of  failing  to  administer  beneficial  treatment  ("regret  of  omission").  The  second  scale  measures  the  regret 
of  administration  of  unnecessary  treatment  ("regret  of  commission").  Using  these  two  scales  we  can  capture  trade-offs 
and  compute  the  threshold  probability  at  which  a  decision  maker  is  indifferent  between  two  alternative  management 
strategies. 

We  employed  the  two  visual  analog  scales  with  typical  100  points  [35-37]anchored  by  no  regret  and  maximal 
regret.  This  is  modeled  after  pain  assessment  limiting  the  maximum  possible  pain  that  a  person  can  experience  [38], 
Accordingly,  we  can  elicit  threshold  probabilities  by  asking  the  physician  to  weigh  the  regret  associated  with  wrong 
decisions  (e.g.  giving  unnecessary  treatment  vs.  failure  to  administer  necessary  treatment)  using  a  numerical  (0  to  100) 
scale.  The  questions  may  be  narrowly  defined  related  to  specific  outcomes  (e.g.,  survival/mortality,  heart  attack  etc.). 
We  should,  however,  note  that  most  treatments  are  associated  with  multiple  dimensions,  some  good  and  some  bad. 
This  is  a  fundamental  reason  why  no  universally  accepted  method  for  assessment  of  decision-makers'  preferences  has 
been  developed  so  far.  It  is  very  difficult,  if  not  impossible,  to  accurately  determine  the  trade-offs  across  multiple 
outcomes  that  can  be  permuted  in  a  number  of  ways.  A  solution  to  this  problem  is  to  capture  the  decision-maker's 
global  or  "holistic"  perception  toward  treatment.  By  asking  questions  about  trade-offs  in  this  way,  we  directly  address 
both  cognitive  mechanisms-  intuitive  and  deliberative-  of  the  decision  process.  This,  in  turn,  can  lead  to  more  accurate 
assessment  of  the  decision  makers'  preferences. 

For  example,  to  elicit  the  physician's  threshold  probability,  we  may  ask  the  following  questions: 
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1.  On  a  scale  0  to  100,  where  0  indicates  no  regret  and  100  indicates  the  maximum  regret  you  could  feel,  how 
would  you  rate  the  level  of  your  regret  if  you  failed  to  provide  necessary  treatment  to  your  patient  (i.e.  did  not 
give  treatment  that,  in  retrospect,  you  should  have  given)?  [Note  that  the  answer  to  this  question  corresponds 
to  the  (U1-U3)  expression  in  equation  1)]. 

2.  On  a  scale  0  to  100,  where  0  indicates  no  regret  and  100  indicates  the  maximum  regret  you  could  feel,  how 
would  you  rate  the  level  of  your  regret  if  you  had  administered  unnecessary  treatment  to  your  patient  (i.e. 
administered  treatment  that,  in  retrospect,  should  have  not  been  given)?  [Note  the  answer  to  this  question 
corresponds  to  the  (U4-U2)  expression  in  equation  1).] 

For  example,  suppose  that  the  physician  answers  60  and  30  to  the  questions  1  and  2,  respectively.  This  means 
that  the  physician  considers  60/30=2  times  worse  to  fail  to  administer  treatment  that  should  have  been  given  than  to 
continue  unnecessary  treatment.  Then,  the  threshold  probability  for  this  physician  is: 


Pt 


1  + 


^1  ~  ^3 

Ua  u? 


f  =  33%. 


Thus,  the  physician  would  be  unsure  as  to  whether  to  treat  or  not  the  patient  if  the  patient's  probability  of  disease  as 
computed  by  the  prediction  model  was  33%.  Thus,  the  recommended  action,  which  is  based  on  elicitation  of  the 
decision-maker  preferences,  is  directly  derived  from  the  underlying  theoretical  model. 


Regret  based  decision  curve  analysis  (DCA) 


Decision-makers  may  be  presented  with  many  alternative  strategies  that  can  be  difficult  to  model.  A  simple,  yet 
powerful  approach  that  is  based  on  experience  of  a  typical  practicing  physician  is  to  compare  the  strategy  based  on 
modeling  with  those  scenarios  when  all  or  no  patient  is  treated.  That  is,  the  clinical  alternatives  to  the  prediction  model 
strategy  is  to  assume  that  all  patients  have  the  disease  and  thus  treat  them  all,  or  to  assume  that  no  patient  has  the 
disease  and  thus  treat  none[25].  In  this  case  the  clinical  dilemma  a  physician  faces  when  considering  treatment  is 
threefold:  (1)  treat  all  the  patients  ("treat  all"),  (2)treat  no  patients  ("treat  none"),  and  (3)  use  a  prediction  model  and 
treat  a  patient  if  p  >  Pt  ("model"). 

The  optimal  decision  depends  on  the  preferences  of  the  decision  maker  as  captured  by  the  threshold 
probability.  We  use  Decision  Curve  Analysis  (DCA)  [24,  25]  to  identify  the  range  of  threshold  probabilities  at  which  each 
strategy  ("treat  all",  "treat  none",  and  "model")  is  of  value.  T raditional  DCA  uses  the  (net  expected)  benefits  associated 
with  each  strategy  to  recommend  the  best  strategy  [24,  25],  In  this  work,  we  consider  that  the  optimal  strategy  is  the 
one  that  brings  the  least  regret  in  case  it  is  proven  wrong,  retrospectively. 

One  view  about  decision  curves  is  that  they  should  not  be  used  in  clinical  practice:  the  researcher  determines 
whether  the  decision  curve  justifies  the  use  of  the  model  in  practice  and  then  makes  a  simple  recommendation  yes  or 
no  as  to  whether  clinicians  should  base  their  decisions  on  the  model  [39],  Another  approach,  which  we  propose  here,  is 
that  threshold  probabilities  obtained  in  clinical  practice  should  be  compared  against  the  decision  curve  to  determine 
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which  strategy  should  be  used  (e.g.  use  a  model,  biopsy  all  men,  biopsy  no-one).  This  might  be  necessary  if  there  is  no 
strategy  with  the  highest  net  benefit  across  the  entire  range  of  reasonable  threshold  probabilities. 

Figure  3,  depicts  the  generalized  decision  tree  describing  all  of  the  alternative  strategies.  By  solving  the  decision 
tree,  we  can  estimate  the  expected  regret  associated  with  each  strategy[23,  26,  28,  31-34],  For  example, 

ERg[Model]  =  p(FJV)(U1  -  U3)  +  (1  -  p)(FP)(l/4  -  U2)  (2) 

Here,  FN  (probability  of  false  negatives)  represents  the  conditional  probability  P(p  <Pt  | D  +)  of  not  treating 
the  patient  who  has  the  disease. 

FP  (probability  of  false  positives)  is  the  conditional  probability  P(p  >  Pt \D  — )  of  treating  the  patient  who  does  not  have 
the  disease. 

Similarly, 


TP  =  1  —  FN  =  P(p  >  Pt\D  +)  (probability  of  true  positives):  Probability  of  treating  the  patient  who  has  the  disease. 

TN  =  1  —  FP  =  P(p  <  Pt\D  — )  (probability  of  true  negatives):  Probability  of  not  treating  the  patient  who  does  not 
have  the  disease. 

After  re-scaling  the  utilities  by  dividing  each  utility  with  the  expression  U1  —  t/3,  and  replacing  Ua  =  —L- ,  we 

U1-(/3  i— 

get  the  expression: 


ERg[Model]  =  p(FiV)  +  (1  -  p)(FP) 


=  PC  1  -  TP)  +  (1  -  p) (FP )  r-^- 


P(p  <  Pt  n  D  +)  +  P(p  >  Pt  n  D  -) : 


(3) 


1-Pt 

For  the  strategies  of  administering  treatment  and  not  administering  treatment,  the  expected  regret  is  derived 


as: 


ERg[Treat  all]  =  (1  -  p)(f/4  -  U2)  =  (1  -  p)  1  ^ p  (4) 

ERg[Treat  none]  —  p(U1  —  U3)  —  p  (5) 

Subtracting  each  of  these  expected  regrets  from  the  expected  regret  of  the  "T reat  none"  (baseline)  strategy  we 
obtain  the  "Net  Expected  Regret  Difference  (NERD)": 

Pt 

NERD[Treat  none, Model]  —  ERg [Treat  none]  —  ERg[Model]  ——  p  —  p(l  —  TP)  —  (1  —  p)(FP) - 

P  1  (6) 
=  pO’P)-(l-p)(FP)r-^- 

P, 

NERD[Treat  none,  Treat  al]  —  ERg[Treat  none]  —  ERg[Treat  all]  —  p  — (1— p)- - —  (7) 

1  —  Pt 

NERD[Treat  none, Treat  none]  —  0  (8) 

Note  that  these  are  exactly  the  same  formulas  as  those  derived  by  Vickers  and  Elkin  [25]  who  employ  the 
expected-utility  model  in  "decision  curve  analysis"  (DCA).  The  regret  based  derivation,  however,  is  mathematically  more 
parsimonious.  The  original  DCA  formulation  required  several  mathematical  manipulations  making  the  simplicity  of  regret 
approach  more  attractive.  In  addition,  as  argued  throughout  the  manuscript,  the  regret  formulation  may  have  additional 
decision-theoretical  advantages  as  it  enables  experiencing  consequences  of  decisions  both  at  the  emotional  (system  1) 
and  cognitive  (system  2)  level[23,  40], 
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In  addition  to  equations  6-8,  we  are  interested  in  the  NERD  between  the  strategies  "Treat  all"  and  "Model": 


NERD  [Treat  all,  Model]  =  ERg  [Treat  all]  -  ERg[Model]  =  (l-p)(7W)  J  -  p(FN)  (9) 

1  —  Pt 

The  NERD  equations  associated  with  each  strategy,  6-8,  can  be  further  reformulated  as  follows[23,  25,  26,  28,  31-34,  41]: 

Pt  #TP  #FP  Pt  Q01 

NERD  =  P{p  >  Pt  n  D  +)  -  P(p  >  Pt  n  D  = - 1  J 

-L  i f-  71  71  1  i f- 

Similarly,  equation  9  can  be  re-written  as: 


NERD  =  (1  -p)(7W) 


1  ~Pt 


p(FN)  —  P(p  <  Pt  n  D  — )  - 


P(p  <  PtnD  +)  = 


#TN  Pt  #FN 
n  1  —  Pt  n 


Equations  10  and  11  above  are  useful  when  calculating  NERD  as  a  function  of  Pt.  The  probabilities  P(p  >Pt  n 
D+,  Pp>PtnD—,  Pp>PtnD+,  and  Pp>PttlD—  a  re  estimated  as  follows: 

•  P(p  >  Pt  fl  D  +)  »  the  number  of  patients  who  have  the  disease  and  for  whom  the  prognostic  probability 

is  greater  than  or  equal  to  Pt  (with  #TP  =  number  of  patients  with  true  positive  results ,  P(p  >  Pt  n  D  +)  « 


— ,  where  n  is  the  total  number  of  patients  in  the  study). 

n 

•  P(p  >  Pt  fl  D  — )  «the  number  of  patients  who  do  not  have  the  disease  and  for  whom  the  prognostic 
probability  of  disease  is  greater  than  or  equal  to  Pt  (with  #FP=  number  of  patients  with  false  positive  results 

UFP 

,P(p>PtnD-)^). 

•  P(p  <  Pt  fl  D  +)  «the  number  of  patients  who  have  the  disease  and  for  whom  the  prognostic  probability 
of  disease  is  less  than  Pt  (with  #TN=  number  of  patients  with  true  negative  results,  P(p  <  Pt  n  D  +)  «  ~~)- 

•  P(p  <  Pt  fl  D  — )  «the  number  of  patients  who  do  not  have  the  disease  and  for  whom  the  prognostic 
probability  of  disease  is  less  than  Pt  (with  #FN=  number  of  patients  with  false  negative  results,  P(p  <  Pt  n 

D  — )  *  — ). 

When  computing  NERD[Treat  none,  Treat  all]  we  assume  that  all  patients  have  the  disease,  thus  #TP  is  the 
number  of  people  who  actually  have  the  disease  and  #FP  is  the  number  of  people  who  do  not  have  the  disease  but  are 
given  treatment.  On  the  other  hand,  when  computing  NERD[Treat  none,  Model ]  from  equation  10  and 
NERD[Treat  all,  Model ]  from  equation  11,  #TP,  #FP,  #TN,  and  #FN  are  computed  for  each  threshold  probability 
assuming  that  a  patient  has  the  disease  if  the  prognostic  probability  is  greater  than  or  equal  to  the  threshold  probability 
and  does  not  have  the  disease,  otherwise. 

NERDs  of  each  of  the  strategies  described  are  plotted  against  different  values  of  threshold  probability.  The  NERD 
values  provide  information  relative  to  decrease  in  regret  when  two  strategies  are  compared  against  each  other  for  a 
given  threshold  probability.  If  NERD=0,  this  means  that  there  is  no  difference  in  the  regret  between  two  strategies: 


NERD[strategy  1, strategy  2]  =  0  <=> 
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ERg  (strategy  1)  —  ERg  (strategy!)  =  0  <=> 

ERg(strategy  1)  =  ERg(strategy  2) 

If  NERD>0,  this  means  that  the  second  strategy  will  inflict  less  regret  than  the  first  strategy,  and  hence  it  is  preferable: 


NERD  [strategy  1,  strategy  2]  >  0  <=> 

ERg (strategy  1)  >  ERg(strategy 2) 

Similarly,  if  NERD<0,  the  first  strategy  represents  the  optimal  decision  among  the  two  strategies: 


NERD[strategy  1,  strategy  2]  <  0  <=> 
ERg  (strategy  1)  <  ERg(strategy2) 

The  algorithm  for  the  Regret  DCA  is  implemented  as  follows: 


(14) 


1.  Select  a  value  for  threshold  probability. 

2.  Assuming  that  patients  should  be  treated  if  p  >  Pt  and  should  not  be  treated  otherwise,  compute  #TP  and  #FP 
for  the  prediction  model. 

3.  Calculate  the  NERD(Treat  none,  Model) using  equation  10. 

4.  Calculate  NERD(Treat  all,  Model)  using  equation  11. 

5.  Compute  the  NERD(Treat  none,  Treat  all) using  equation  10  where  #TP  is  the  number  of  patients  having  the 
disease  and  #FP  is  the  number  of  patients  without  disease  who  got  treatment. 

6.  Repeat  steps  1  -  6  for  a  range  of  threshold  probabilities. 

7.  Graph  each  NERD  calculated  in  steps  3-5  against  each  threshold  probability. 

Based  on  the  Regret  DCA  methodology,  the  optimal  decision  at  each  threshold  probability  is  derived  by 
comparing  each  pair  of  strategies  through  their  corresponding  NERDs  according  to  the  transitivity  principle  (i.e.,  if  A>B, 
B>C  then  A>C).  Thus,  if  NERD(strategyl,  strategy 2)  >  NERD  (strategy!,  strategy 3)  >  0  then  strategy  2  is  better 
than  strategy  1,  and  strategy  3  is  better  than  strategy  2.  Therefore,  strategy  3  is  the  optimal  strategy. 


Acceptable  Regret 


No  decision  model  can  guarantee  that  the  recommended  strategy  will  be  the  correct  one.  Therefore,  we  can 
always  make  a  mistake  and  recommend  treatment  we  should  not  have,  or  fail  to  recommend  treatment  we  should  have 
administered  [42],  However,  there  are  situations  where  the  regret  resulting  from  a  wrong  decision  will  be  tolerable. 
These  situations  are  best  described  under  the  notion  of  acceptable  regret  [26,  28,  31],  Formally,  acceptable  regret,  Rg0l 
is  defined  as  the  portion  of  utility  a  decision  maker  is  willing  to  lose/sacrifice  when  he/she  adheres  to  a  decision  that 
may  prove  wrong  [26,  28,  31,  32],  For  example,  a  physician  may  regret  administering  unnecessary  treatment  to  a  patient 
but  he/she  can  "still  live  with"  the  consequences  of  this  decision  if  she/he  judged  them  to  be  trivial  or  inconsequential. 

We  assume  that  there  is  a  linear  relationship  between  the  value  of  acceptable  regret  and  the  benefits  of 
receiving  treatment  as  well  as  the  harms  of  receiving  unnecessary  treatment.  This  is  a  reasonable  assumption  because 
acceptable  regret  is  expected  to  operate  within  a  narrow  range,  at  the  lower  or  the  upper  end,  of  the  probability  scale. 
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We  define  acceptable  regret  in  terms  of  benefits  of  treatment,  Rgb,  as[43]  the  percentage  (rb)  of  benefits  (U1  —  f/3)  the 
decision  maker  is  willing  to  forgo  if  his/her  decision  NOT  to  treat  was  wrong: 


Rgo  =  Rgb  =  rbB  =  rb(u1  -  u3)  (15) 

Alternatively,  we  define  acceptable  regret  in  terms  of  harms  of  unnecessary  treatment,  Rghl  as[43]  the  percentage  (rh) 

of  harms  (f/4  —  U2 )  the  decision  maker  is  willing  to  incur  if  his/her  decision  of  treating  was  wrong: 


Rgo  =  Rgh  =rhH  =  rh  (f/4  -  U2 )  (16) 

We  use  the  concept  of  acceptable  regret  to  further  refine  the  conditions  under  which  the  decision  maker  is 
indifferent  between  two  strategies.  Recall  that  these  conditions  have  been  initially  captured  in  terms  of  threshold 
probability,  which  does  not  incorporate  the  sense  of  tolerable  losses.  Thus,  we  proceed  with  the  following  definition: 

Two  strategies  are  considered  equivalent  in  regret  (e.g.  will  bring  the  same  regret  to  the  decision  maker  if  they  are 
proven  wrong,  in  retrospect),  if  the  absolute  value  of  their  net  expected  regret  difference  (NERD)  is  less  than  or  equal  to 
a  predetermined  amount  of  acceptable  regret  Rg0.  In  other  words,  there  is  no  difference  between  choosing  the  strategy 
"treat  all"  or  "treat  none"  in  terms  of  regret  if: 

\NERD(Treat  none,  Treat  all)\  <  Rg0  (17) 

Similarly,  the  strategies  "model"  and  "treat  none"  are  equivalent  in  regret  if: 

\NERD (Treat  none,  Model^  <  Rg0  (18) 

and  the  strategies  "model"  and  "treat  all": 

\NERD (Treat  all, Model)\  <  Rg0  (19) 

The  acceptable  regret,  Rg0,  can  be  computed  using  any  of  the  two  definitions  described  in  equations  15  and  16. 

We  can  also  use  equations  15  and  16  to  identify  the  prognostic  probabilities  at  which  the  decision  maker  would 
not  regret  the  decision  to  which  he/she  is  committed  even  if  that  decision  may  prove  wrong.  For  instance,  we  are 
typically  interested  in  the  prognostic  probability  above  which  a  physician  would  commit  to  the  decision  to  treat  a 
patient,  and  the  probability  below  which  he/she  would  not  to  treat  a  patient  without  feeling  undue  consequences  of 
these  decisions[28].  In  other  words,  we  are  looking  for  the  probabilities  for  which  ERg(Treat  all )  <  Rgh,  and 
ERg(Treat  none )  <  Rgb.  Solving  the  inequalities  using  equations  4,  5,  15,  and  16  and  after  scaling  Rg0  by  (U1  —  f/3), 
we  obtain 


Ptreat  all  =  1  “  rh  (20) 

where  Ptreat  au  is  the  prognostic  probability  above  which  the  physician  would  tolerate  giving  treatment  that  may  prove 

unnecessary.  Similarly, 


Ptreat  none  T)  (21) 

represents  the  prognostic  probability  below  which  the  physician  would  comfortably  withhold  treatment  that  may  prove 
beneficial,  in  retrospect. 

Note  that  equations  20  and  21  express  acceptable  regret  in  terms  of  probabilities  while  equations  17-19  define 
it  in  terms  of  NERD.  Hence,  the  outputs  of  these  equations  are  not  the  same;  rather,  they  complement  each  other. 

Elicitation  of  acceptable  regret 

In  most  cases  the  decision  maker  does  not  have  a  complete  understanding  of  benefits  lost  or  harms  inflicted  and 
cannot  assign  a  precise  number  to  them.  For  this  reason,  we  do  not  suggest  inquiring  directly  about  the  value  of  r. 
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Instead,  we  propose  eliciting  r  through  the  decision-maker's  responses  to  specific  clinical  scenarios.  For  example,  we 
propose  the  following  approach  (see  Table  1  for  the  specific  example): 

Assume  that  you  have  100  patients  with  the  same  probability  of  disease  as  the  patient  you  are  currently 
treating.  You  need  to  decide  whether  each  of  these  patients  should  receive  treatment  or  not.  Since  no  prediction  model  is 
100%  accurate,  it  is  expected  that  you  will  make  some  mistakes  in  your  treatment  recommendations  (e.g.  you  may 
recommend  treatment  to  a  patient  who  does  not  need  it,  or  fail  to  recommend  treatment  to  a  patient  who  needs  it). 

1.  We  are  now  interested  in  knowing  your  tolerance  toward  administering  unnecessary  treatment  i.e.  we  want  to 
learn  what  the  magnitude  of  the  unavoidable  error  you  can  live  with  is  by  inflicting  potentially  harmful 
treatment  on  a  patient.  Note  that  if  you  say  that  your  acceptable  regret  is  zero,  this  means  that  you  can  only 
make  decision  if  you  absolutely  certain  that  your  recommendation  is  correct. 

Out  of  the  number  (100-y)  of  patients  who  should  have  not  received  treatment,  how  many  patients  would  you 
tolerate  treating?  (The  answer  is  used  to  compute  rft). 

2.  We  are  interested  in  knowing  your  tolerance  toward  failing  to  provide  necessary  treatment  i.e.  we  want  to  learn 
what  the  magnitude  of  unavoidable  error  you  can  live  with  is  by  forgoing  potentially  beneficial  treatment.  Note 
that  if  you  say  that  your  acceptable  regret  is  zero,  this  means  that  you  can  only  make  decision  if  you  absolutely 
certain  that  your  recommendation  is  correct. 

Out  of  the  number  (100-x)  of  patients  who  should  have  been  treated,  how  many  patients  would  you  tolerate  not 
treating?  (The  answer  is  used  to  compute  rb ). 

It  is  unnecessary  to  ask  the  decision  maker  to  answer  both  questions.  We  suggest  asking  only  the  question 
related  to  the  recommendation  the  physician  is  about  to  make  e.g.  if  the  recommendation  is  about  administering 
treatment,  then  the  decision  maker  should  be  asked  the  second  question,  while  if  it  is  about  not  giving  treatment,  then 
he/she  can  ask  the  first  question. 

The  value  of  acceptable  regret  is  plotted  in  the  regret  DCA  graph  to  visually  facilitate  the  decision  making 
process.  At  a  specific  threshold  probability  all  strategies  for  which  \NERD\  <  Rg0  are  considered  equivalent  in  regret, 
according  to  the  definition  in  the  previous  section. 

Example 

We  will  employ  a  prostate  cancer  biopsy  example  to  demonstrate  the  applicability  of  our  approach.  Prostate 
cancer  biopsy  is  an  invasive  and  uncomfortable  procedure,  which  can  be  painful  and  is  associated  with  a  risk  of  infection. 
However,  it  is  often  necessary  for  diagnosis  of  prostate  cancer,  one  of  the  leading  causes  of  cancer  death  in  men. 

Men  are  typically  biopsied  for  prostate  cancer  if  they  have  an  elevated  level  of  prostate-specific  antigen  (PSA). 
However,  most  men  with  a  high  PSA  do  not  have  prostate  cancer.  This  has  led  to  the  idea  that  statistical  models  based 
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on  multiple  predictors  (PSA,  age,  family  history,  other  markers)  might  be  used  to  predict  biopsy  outcomes  and  hence  aid 
biopsy  decisions  for  individual  patients.  A  physician  seeing  a  patient  with  an  elevated  PSA  has  three  possible  options:  go 
for  biopsy,  refuse  biopsy  or  look  up  his  probability  in  a  statistical  model  and  then  make  a  decision. 

We  utilize  an  unpublished  statistical  model  that  computes  probability  of  cancer  based  on  the  dataset  described 
in  [44]  to  compare  each  of  these  options.  Following  the  algorithm  described  in  the  regret  DCA  section,  we  generate  the 
decision  curves  depicted  in  Figure  4.  This  figure  is  used  to  determine  the  optimal  strategy  for  different  values  of 
threshold  probability.  The  optimization  procedure  is  implemented  in  three  steps  where  the  strategies  in  each  NERD  are 
compared  to  each  other  as  in  equations  12- 14  at  a  specific  threshold  probability.  For  example,  at  threshold  probability 
15%: 

1.  NERD  (biopsy  none,  model )  >  0  therefore,  the  model  is  preferred  to  the  strategy  biopsy  none. 

2.  NERD(biopsy  none,  biopsy  all )  >  0  therefore,  the  strategy  biopsy  all  is  preferred  to  the  strategy  biopsy  none. 

3.  NERD  (biopsy  all,  model )  >  0  therefore,  the  model  is  preferred  to  the  strategy  biopsy  none 
Consequently,  "model"  corresponds  to  the  optimal  strategy. 

Repeating  the  same  procedure  for  all  threshold  probabilities,  we  can  see  that  deciding  based  on  the  statistical 
model  is  the  optimal  strategy  (i.e.  results  in  the  minimum  expected  regret)  for  threshold  probabilities  between  8%  and 
43%.  For  threshold  probabilities  between  42%  and  95%,  the  optimal  strategy  is  to  biopsy  no  patients,  while  for  0%  to  8% 
both  model  and  biopsy  all  strategies  are  optimal. 

To  interpret  these  results,  we  have  to  consider  how  a  typical  physician  values  the  harms  of  a  false  negative 
(missing  a  cancer)  and  a  false  positive  (an  unnecessary  biopsy)  result.  If  regret  associated  with  unnecessary  biopsy  is  felt 
to  be  worse  than  missing  cancer,  then  according  to  equation  1,  the  threshold  probability  is  greater  than  50%.  However, 
it  is  unlikely  that  a  physician  would  consider  an  unnecessary  biopsy  to  be  worse  than  missing  a  cancer,  so  the  threshold 
probability  for  biopsy  must  be  less  than  50%.  Thus,  a  reasonable  range  of  threshold  probabilities  might  indeed  be 
between  8%  -  43%  as  suggested  by  our  model.  As  the  model  is  superior  across  this  entire  range,  we  can  conclude  that, 
irrespective  of  the  physician's  exact  preferences,  making  a  biopsy  decision  based  on  the  statistical  model  will  lead  to 
lower  expected  regret  than  an  alternative  such  as  biopsying  all  or  no  men.  Based  on  discussions  with  clinicians,  we 
believe  that  a  reasonable  range  of  threshold  probability  is  10%  -  40%.  As  the  regret  associated  with  the  model  strategy  is 
lowest  across  this  entire  range,  we  can  recommend  use  of  the  model.  Nonetheless,  we  do  not  have  a  complete  sample 
of  all  physician  preferences  and  it  is  possible  that  a  physician  may  have  a  probability  outside  of  this  range. 

To  illustrate  the  applicability  of  the  acceptable  regret  model,  assume  that  the  value  of  acceptable  regret  for 
forgoing  the  benefits  of  biopsy  (equation  15)  is  equal  to  +0.01.  Consider  the  case  that  the  decision  maker's  threshold 
probability  is  equal  to  20%.  According  to  Figure  4,  the  optimal  strategy  should  be  suggested  by  the  statistical  model. 
However,  we  see  that 


\N ERD (treat  none,  mo del)\  <  0.01 

which  means  that  the  strategies  "biopsy  none"  (biopsy  no  patients)  and  "model"  are  equivalent  in  regret.  Therefore,  the 
prediction  model  does  not  offer  any  better  information  and  thus,  it  can  be  disregarded. 

Tablel  describes  the  overall  decision  process  regarding  prostate  cancer  biopsy.  The  process  begins  with 
elicitation  of  the  threshold  probability  from  the  treating  physician  and  continues  with  evaluation  of  the  available 
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strategies  based  on  regret  DCA  (Figure  4).  Based  on  the  optimal  strategy  suggested  by  the  regret  DCA,  the  cancer 
probability  is  estimated.  Finally,  the  optimal  strategy  is  verified  under  the  concept  of  acceptable  regret.  The  normal  font 
text  corresponds  to  the  author  comments.  The  text  in  italic  and  bold  font  corresponds  to  questions  to,  and  answers  from 
the  physician  respectively.  The  underlined  text  is  notes  to  the  reader.  We  demonstrate  the  applicability  of  our  approach 
using  hypothetical  answers  from  two  physicians. 

Discussion 

Currently,  there  is  no  agreed  upon  method  for  how  preferences  regarding  multiple  objectives  that  typically  go  in 
opposite  directions  (i.e.  most  medical  interventions  are  associated  both  with  benefits  and  harms)  should  be  elicited.  We 
have  presented  and  demonstrated  an  approach  to  decision  making  based  on  regret  theory  and  decision  curve  analysis. 
The  approach  presented  in  this  paper  relies  on  the  concept  of  the  threshold  probability  at  which  a  decision  maker  is 
indifferent  between  strategies,  to  suggest  the  optimal  decision  [27,  29,  30],  Unlike  the  approaches  described  in  the 
classic  threshold  papers  [27,  29,  30],  our  approach  is  based  on  the  notion  that  the  value  of  threshold  probability  is 
clearly  subjective  and  depends  on  the  personal  preferences  of  the  decision  maker.  We  elicit  threshold  probabilities 
based  on  the  regret  one  may  feel  in  case  that  the  chosen  strategy  is  proven  wrong,  in  retrospect.  Although  one  can 
narrow  down  the  approach  to  specific  medical  outcomes,  we  believe  that  eliciting  preferences  in  a  global,  holistic  way  is 
more  useful  if  our  approach  is  to  be  used  in  the  actual  practice. 

We  believe  that  the  model  described  here  has  a  direct  practical  application  in  overcoming  many  difficulties 
related  to  linking  evidence  with  patient's  preferences  to  arrive  at  the  optimal  decision-  the  issues  that  plagued  the  field 
of  decision-making.  The  problem  of  eliciting  preferences  and  integrating  them  in  a  coherent  decision  is  not  a  simple  one. 
We  argue  that  the  approach  we  are  advocating  here  represents  a  contribution  to  the  field  of  decision  making,  be  should 
not  be  seen  as  the  panacea  to  medical  decision  making.  However,  we  anticipate  our  methodology  to  be  suitable  for 
medical  decision  primarily  associated  with  trade-offs  between  quality  and  quantity  of  life. 

Over  that  last  couple  of  decades,  many  attempts  have  been  made  to  develop  the  best  method  to  take  these 
considerations  in  real-life  settings.  Unfortunately,  as  explained,  no  approach  has  succeeded  [35],  We  believe  that  the 
reason  for  this  is  that  most  approaches  to  elicit  decision  maker's  preferences  as  well  as  to  help  improve  decision-making 
have  relied  on  a  rational  framework  based  on  expected  utility  theory[21j.  However,  modern  cognitive  theories  (within 
so  called  dual-processing  theory)  have  convincingly  demonstrated  that  human  decisions  rely  both  on  intuition  (system 
1)  and  analytical,  deliberative  process  (system  2)  in  balancing  risks  and  benefits  in  the  decision-making  process  [22 , 40 , 
45],  We  believe  that  rational  decision-making  should  take  into  account  both  formal  principles  of  rationality  and  human 
intuition  about  good  decisions  [46,  47],  The  key  is  to  preserve  rational  framework,  while  allowing  anticipation  of  the 
effect  of  decision  on  emotions  (while  avoiding  biases  associated  with  intuitive  thinking)  [40],  One  way  to  accomplish  this 
is  to  use  the  cognitive  emotion  of  regret  to  serve  as  a  link  between  system  1  (i.e.  intuitive  system)  and  system  2  (i.e. 
deliberative,  analytical  cognitive  system).  By  anticipating  consequences  of  our  actions  and  circumstances  under  which 
we  can  live  with  our  mistakes,  we  bring  together  both  aspects  of  cognition  that  may  lead  to  better  and  more  satisfactory 
decision-making. 

Specifically,  we  argue  that  eliciting  people's  preferences  using  regret  theory  may  be  superior  to  using  traditional 
utility  theory  because  regret  forces  decision-makers  to  explicitly  consider  consequences  of  decisions.  We  have 
previously  shown  that  we  can  always  make  errors  in  decision-making:  recommend  treatment  that  does  not  work,  or  fail 
to  recommend  treatment  that  does  [26],  Therefore,  we  reformulated  DCA  from  the  regret  theory's  point  of  view. 
Furthermore,  it  has  been  shown  that  the  expected  utility  theory  is  often  violated  to  minimize  anticipated  regret  [33, 
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34], In  addition,  there  is  substantial  evidence  that  medical  decision  making  aims  to  minimize  regret  associated  with 
wrong  decisions  [48-50], 

Moreover,  while  descriptive,  normative,  and  prescriptive  theories  [17]  tend  to  evaluate  individual  outcomes,  the 
approach  presented  here  evaluates  all  of  the  outcomes  in  a  holistic  manner.  Our  approach  is  consistent  with  Reyna's 
"gist"  or  "fuzzy  trace  theory"  in  which  the  decision-maker  characterizes  gist  of  each  outcome  to  arrive  at  a  given 
decision  [51],  For  example,  consider  that  a  decision  maker  is  provided  with  a  list  of  harms  and  benefits  associated  with 
each  decision,  as  it  is  currently  recommended  by  the  practice  guidelines  panels  [52],  In  traditional  theories,  the  decision 
maker  evaluates  a  treatment  strategy  by  reasoning  on  each  of  the  harms  and  benefits  associated  with  a  given  strategy. 
This,  as  discussed  above,  would  mean  integration  of  all  multiple  outcomes  that  often  go  in  different  directions  typically 
within  limited  time-frame.  Due  to  the  complexity  of  these  decisions,  however,  this  approach  overwhelms  the  decision 
maker  as  our  brain  capacity  is  limited.  The  regret  DCA  methodology  quantifies  the  global  attitudes  of  the  decision  maker 
towards  a  specific  strategy  without  requiring  separate  reasoning  for  each  of  the  harms  and  benefits.  This  holistic 
assessment  occurs  within  the  dual  processing  cognitive  system,  which  evaluates  collectively  the  harms  and  the  benefits 
associated  with  each  treatment  alternative.  By  assessing  trade-offs  through  both  cognitive  mechanisms-intuitive  and 
deliberative-  we  believe  that  we  can  assess  decision  makers'  preferences  more  accurately. 

In  general,  since  our  method  relies  on  the  elicitation  of  threshold  probability  we  recommend  using  our 
methodology  for  every  patient.  As  every  patient's  values  are  different  the  threshold  probability  should  indeed  be 
patient-  specific.  For  example,  a  physician  may  act  "aggressively"  for  a  young  patient  who  is  the  father  of  two  underage 
kids  and  less  aggressively  for  an  older  patient.  However,  in  the  cancer  biopsy  example,  it  is  expected  that  most  of  the 
patients  should  present  with  similar  characteristics  and  therefore  most  physicians  would  settle  in  a  small  area  of 
threshold  probabilities.  In  this  case  repeating  the  elicitation  process  for  every  patient  would  be  impractical. 

Nevertheless,  this  is  an  empirical  question  worthy  of  further  investigation  as  alluded  above. 

Our  approach  may  help  reconcile  formal  principles  of  rationality  and  human  intuitions  about  good  decisions 
that  may  better  reflect  "rationality"  in  medical  decision-making  [21,  32,  46,  47],  We  hope  that  our  theoretical  work  will 
stimulate  empirical  testing  of  the  concepts  outlined  in  this  paper.  Toward  this  end,  we  are  currently  working  on 
developing  a  prescriptive  computerized  decision-support  system  to  facilitate  the  application  of  the  model  described 
herein.  Such  a  system  is  expected  to  be  user  friendly  with  built-in  automatic  manipulation  of  the  complex  calculations 
that  may  be  off-putting  to  many  users.  We  hope  to  report  on  testing  of  our  system  in  the  near  future. 

Conclusions 

We  have  presented  a  decision  making  methodology  that  relies  on  regret  theory  and  decision  curve  analysis  to 
assist  physicians  in  choosing  between  appropriate  health  care  interventions.  Our  methodology  utilizes  the  cognitive 
emotion  of  regret  to  determine  the  decision  maker's  preferences  towards  available  strategies  and  DCA  to  suggest  the 
optimal  decision  for  the  specific  decision  maker.  We  believe  that  our  approach  is  suitable  for  those  clinical  situations 
when  the  best  management  option  is  the  one  associated  with  the  least  amount  of  regret  (e.g.  diagnosis  and  treatment 
of  advanced  cancer,  etc). 

As  with  any  other  novel  theoretical  work,  our  approach  has  its  limitations.  First,  it  has  not  been  empirically 
tested  in  a  clinical  setting.  However,  we  are  in  the  process  of  developing  the  appropriate  decision  support  tools  to  bring 
our  model  into  clinical  practice  and  evaluate  its  usefulness  with  actual  physicians  and  patients.  Second,  the  methodology 
presented  is  appropriate  for  single  point  decision  making.  Further  investigation  is  required  to  determine  the  application 
of  regret  theory  to  decisions  that  re-occur  over  time.  Finally,  we  assume  that  there  is  only  one  decision  maker  involved 
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in  the  decision  process.  Nevertheless,  our  plan  for  future  work  includes  extending  our  methodology  to  shared  decision¬ 
making  that  will  include  both  physician  and  patient  in  the  decision  process  and  investigate  whether  in  practice  there  is  a 
difference  between  preferences  and  choices  made  by  physicians  and  their  patients. 

We  summarize  the  contribution  presented  in  this  paper  as  follows: 

1.  We  propose  a  novel  method  for  eliciting  decision  makers'  preferences  towards  treatment  administration. 
Contrary  to  traditional  methodologies  on  eliciting  preferences,  our  method  considers  the  consequences  of 
potential  mistakes  in  decisions.  We  propose  a  dual  visual  analog  scale  to  capture  errors  of  omission  and 
errors  of  commission  and,  therefore,  evaluate  the  trade-offs  associated  with  each  of  the  available  strategies. 

2.  We  have  reformulated  DCA  from  the  regret  theory  point  of  view.  Our  approach  is  intuitively  more  appealing 
to  a  decision  maker  and  should  facilitate  decision  making  particularly  in  those  clinical  situations  when  the 
best  management  option  is  the  one  associated  with  the  least  amount  of  regret. 

3.  Finally,  we  utilize  the  concept  of  acceptable  regret  to  identify  the  circumstances  under  which  a  decision 
maker  tolerates  a  wrong  decision. 


We  envision  facilitation  of  the  decision  process  in  clinical  settings  through  a  computerized  decision  support 
system  available  at  the  point  of  care.  In  fact,  we  are  in  the  process  of  developing  such  a  system  and  hope  to  report 
about  it  soon. 


List  of  abbreviations 

DCA:  Decision  Curve  Analysis 
NERD:  Net  Expected  Regret  Difference 
VAS:  Visual  Analog  Scale 


Glossary 


V 

Prognostic  probability 

Pt 

Threshold  probability 

D  +  /  D- 

The  patient  has/  does  not  have  the  disease 

Ut 

Utility  corresponding  to  outcome  i 

R.q(x) 

Regret  associated  with  the  action  x 

Rx  + /Rx  — 

Treatment  /  No  treatment 

Ui  -u. 

Consequences  of  not  administering  treatment  where  indicated 

U 4  -  U? 

Consequences  of  unnecessarily  administering  treatment 

ERg  (action) 

Expected  regret  associated  with  an  action 
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TP,  TN,  FP,  FN 

Conditional  probabilities 

#TP,  #TN,  #FP,  #FN 

Number  of  TP,  TN,  FP,  FN  patients 

n 

Number  of  patients 

NERD  ( actionl ,  action2) 

Net  expected  regret  difference  between  actions  1  and  2 

R.q  o 

Acceptable  regret 

Rdh 

Acceptable  regret  as  defined  in  terms  of  loses  in  benefits  due  to 
forgoing  treatment 

Rdh 

Acceptable  regret  as  defined  in  terms  of  harms  due  to  undergoing 
unnecessary  treatment 

rb/rh 

Percentages  of  the  benefits/harms  a  decision  maker  is  willing  to 
lose/incur  in  case  of  a  wrong  decision 

D 

1  treat  all 

The  prognostic  probability  above  which  the  decision  maker  would 
tolerate  recommending  unnecessary  treatment. 

p 

1  treat  none 

The  prognostic  probability  below  which  the  decision  maker  would 
tolerate  not  recommending  treatment. 
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Figure  1.  Decision  tree  for  treatment  distribution. 

In  this  figure,  p  =  P{D  +)  is  the  probability  associated  with  the  presence  of  a  disease;  1  —  p  =  P(D  —)  is  the  probability 
associated  with  the  absence  of  the  disease;  Ui,  i  e  [1,4],  are  the  utilities  corresponding  to  each  outcome.  Note  that  we 
use  the  term  "treatment"  in  the  generic  sense  of  health  care  intervention,  which  may  indicate  therapy,  procedure,  or  a 
diagnostic  test. 
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P>P, 


ORg(Rx+,D+)=0 


1-P 


-ORg(Rx+,  D-)=U4-U2 


Do  not  treat 
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P>Pt 


ORg(Rx-,D+)=U,-U3 


1-P 


-ORg(Rx-,D-)=0 


Figure  2.  Regret  model  of  decision  tree  treating  a  patient. 

In  this  figure,  p:  probability  of  having  the  disease;  1-p:  probability  of  not  having  the  disease;  Pt:  threshold  probability  for 
treatment;  Rg:  regret  associated  with  wrong  decisions;  Rx-:  no  treatment;  Rx+:  treatment;  D+:  disease  is  present;  D-: 
disease  is  absent.  For  example,  Rg(Rx+,  D-):  regret  associated  with  the  error  of  treating  the  patient  who  did  not  have  the 
disease. 


Treat  all 


-OUi  ,  Rg(Rx+,D+)=0 
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-OU2  ,  Rg(Rx+,  D-)=U4-U2 
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Treat 


-OUi  ,  Rg(Rx+,D+)=0 
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-OU2  ,  Rg(Rx+  D-)=U4-U2 
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-OU3  ,  Rg(Rx-.D+)=U1-U3 
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Figure  3.  Generalized  decision  tree  for  administration  of  treatment. 

In  this  figure,  p:  probability  of  having  the  disease;  1-p:  probability  of  not  having  the  disease;  Rg:  regret  associated  with 
wrong  decisions;  Rx-:  no  treatment;  Rx+:  treatment;  D+:  disease  is  present;  D-:  disease  is  absent. 
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Threshold  probability 

- NERD(b»psy  none,  biopsy  none)  NERD(b»psy  none,  biopsy  all) 

- NERD(b»spy  none,  model)  . NERD(b»psy  al.  model) 

-  -  Acceptable  regret 

Figure  4.  Regret  DCA  regarding  biopsy  to  detect  prostate  cancer. 

Thin  line:  biopsy  all  patients;  solid  line:  biopsy  no  patients;  dashed  line:  prediction  model.  The  optimal  strategy  is  derived 
by  the  comparison  of  each  pair  of  strategies  from  all  NERDs  as  per  equations  12-14.  The  statistical  model  is  the  optimal 
strategy  for  threshold  probabilities  between  8%  and  42%.  For  threshold  probabilities  between  43%  and  95%,  the  optimal 
strategy  is  to  biopsy  no  patients,  while  for  0%  to  8%  both  "model"  and  "biopsy  all"  strategies  are  optimal.  The  lines  of 
acceptable  regret  denote  the  regret  area  in  which  different  strategies  are  eguivalent.For  example,  at  threshold 
probability  egual  to  20%,  the  optimal  strategy  is  acting  based  on  the  prognostic  model.  However, 

NERD  (biopsy  none,  model )  is  below  the  acceptable  regret  line  which  indicates  that  the  strategies  "biopsy  none"  and 
"model"  are  eguivalent  in  regret.  Therefore  the  optimal  strategy  is  to  biopsy  no  patients  as  the  use  of  model  is  deemed  to 
be  superfluous.  Similarly, at  threshold  probability  egual  to  15%,  the  optimal  strategy  is  to  act  based  on  the  model  and  the 
strategies  "biopsy  all"  and  "biopsy  none"  are  eguivalent  in  regret.  Finally,  at  threshold  probability  egual  to  9%,  the 
optimal  strategies  are  both  "model"  and  "biopsy  all".  However,  since  NERD  (biopsy  all,  model )  is  below  the  acceptable 
regret,  the  strategies  "biopsy  all"  and  "model"  are  eguivalent  in  regret.  Therefore  the  optimal  strategy  is  to  biopsy  all 
patients. 


Tables 

Table  1.  Example:  Decision  process  regarding  prostate  cancer  biopsy. 

The  normal  font  text  corresponds  to  the  author  comments.  The  text  in  italic  and  bold  font  corresponds  to  guest  ions  to 
and  answers  from  the  physicians,  respectively.  The  underlined  text  is  notes  to  the  reader.  We  examine  the  problem  from 
the  perspective  of  two  different  (hypothetical)  physicians. 

1.  Interview  with  the  physician  to  elicit  his/her  threshold  probability. 
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a.  On  the  scale  0  to  100,  where  0  indicates  no  regret  and  100  indicates  the  maximum  regret  you 
could  feel,  how  would  you  rate  your  level  of  regret  if  you  failed  to  provide  necessary  treatment? 
Physician  #1  answer:  50,  Physician  #2  answer:  70.  This  value  corresponds  to  U1  —  U 3  from 
equation  1. 

b.  On  the  scale  0  to  100,  where  0  indicates  no  regret  and  100  indicates  the  maximum  regret  you 
could  feel,  how  would  you  rate  your  level  of  regret  if  you  administered  unnecessary >  treatment? 
Physician  #1: 10,  Physician  #2:  60.  This  value  corresponds  to  UA  —  U1  from  equationl. 

The  threshold  probability  is  equal  to  (equation  1):  Physician  #1:  16%,  Physician  #2:  46%. 

2.  Using  the  graph  in  Figure  4,  identify  the  optimal  strategy  for  the  computed  threshold  probability. 

Physician  #  1 :  For  threshold  probability  equal  to  16%.  the  optimal  decision  is  derived  by  solving  the 
inequalities  (Figure  4.  equations  12-14): 

1.  NERD(biopy  all, model)  >  0.  the  strategy  “model"  is  better  than  the  strategy  “biopsy  all" 

2.  NERD  (biopsy  none,  model)  >  0,  the  strategy  ‘‘model”  is  better  than  the  strategy  “biopsy  none” 

3.  NERD  (biopsy  none,  biopsy  all)  >  0,  the  strategy  “biopsy  all”  is  better  than  “biopsy  none”. 

Therefore,  the  optimal  strategy  is  the  “model”  which  corresponds  to  biopsy  based  on  the  probability  of 
cancer  predicted  by  the  statistical  model.  The  next  step  will  be  to  compute  the  patient’s  probability  of 
cancer  and  contrast  it  with  the  threshold  probability. 

Physician  #2:  For  threshold  probability  equal  to  46%.  the  optimal  decision  would  be  the  “biopsy  none” 
strategy.  In  this  case,  even  though  computing  the  probability  of  cancer  will  not  affect  the  physician’s 
decision,  it  will  help  identify  the  circumstances  under  which  the  physician  would  tolerate  unnecessary 
biopsy  of  the  patient. 

3.  Compute  the  cancer  probability  for  the  specific  patient  based  on  the  statistical  model. 

a.  If  the  cancer  probability  is  greater  than  or  equal  to  the  threshold  probability,  then  the  surgeon 
should  biopsy  the  patient. 

b.  If  the  cancer  probability  is  less  than  the  threshold  probability,  then  the  surgeon  should  not  biopsy 
the  patient. 

Let  us  assume  that  the  probability  of  cancer  for  the  specific  patient  is  equal  to  20%.  The  threshold 
probability  for  Physician  #1  is  16%  (as  computed  in  step  1).  In  this  case.  Physician  #1  considers 
recommending  biopsy.  As  noted  in  step  2b.  the  best  strategy  for  Physician  #2  is  recommending  not  to 
biopsy  any  patients  regardless  their  probability  of  cancer. 

4.  Elicitation  of  the  level  of  acceptable  regret. 

Assume  that  you  have  100  patients,  all  with  probability  of  cancer  equal  to  20%  (the  same  as  your 
patient).  This  means  that  out  of  100  patients,  20  patients  will  have  cancer  while  80  will  not  have 
cancer.  You  need  to  decide  whether  each  of  these  patients  should  undergo  biopsy  or  not.  Since  no 
prediction  model  is  100%  accurate,  it  is  expected  that  you  will  make  some  mistakes  in  your 
recommendations  (e.g.  you  may  recommend  biopsy  to  a  patient  who  does  not  need  it,  or  fail  to 
recommend  biopsy  to  a  patient  who  may  need  it). 
a.  The  physician  considers  biopsy  (Physician  #1): 

Out  of  the  20  patients  who  should  be  biopsied,  for  how  many  patients  would  you  tolerate  not 
recommending  a  necessary  biopsy?  1. 

This  answer  corresponds  to  n,  =  —  =  0.05  and  acceptable  regret 


Rgb  —  rb(U1  —  U3 )  =  0.05  *  0.5  =  0.025.  The  optimal  strategy  at  Pt  —  16%  is  to  use  the 
statistical  model  (Figure  4),  For  P,  —  16%  and  Rgh  —  0.025  all  NERDs  are  greater  than 
acceptable  regret,  thus  the  optimal  strategy  remains  the  statistical  model. 
b.  The  physician  does  not  consider  biopsy  (Physician  #2). 

Out  of  the  80  patients  who  should  not  undergo  biopsy,  for  how  many  patients  would  you  tolerate 

recommending  an  unnecessary  biopsy?  40. 

40 

The  answer  provided  by  the  Physician  #2  corresponds  to  rh  —  —  =  0.50  and  acceptable  regret 

80 

Rgh  =  rh  (U4  -  U2)  =  0.5  *  0.6  =  0.3, 

The  optimal  strategy  for  P,  =  46%  is  to  biopsy  no  patients  (Figure  4).  Also,  for  Pt  —  46%  and 

Rgh  =  0.3,  we  have:  \NERD(bioysy  none,  biopsy  all) I  =  | — 0.639 1  >  Rghl 

|(V  ERD  (biopsy  none,  model)]  —  |  —  0.003 1  <  Rph  and  NERD  (biopsy  all,  model)  — 

0.6364  >  Rgh.  This  means  that  the  strategies  “biopsy  none”  and  “moder  are  equivalent  in  regret. 
In  practical  terms  no  additional  effort  is  justified  for  using  the  statistical  model. 

5.  Based  on  equations  20  and  21.  we  can  determine  the  prognostic  probabilities  above  and  under  which 

the  physician  would  tolerate  performing  an  unnecessary  biopsy,  or  not  to  do  so  when  he  should  have 

done  it. 

a.  Physician  #1  considers  recommending  biopsy  to  his/her  patient.  Based  on  equation  21,  the 
physician  would  tolerate  not  recommending  a  biopsy  for  any  prognostic  probability  below 

Ptreat  none  T)  5%. 

b.  Physician  #2  considers  not  recommending  biopsy  to  his/her  patient.  Based  on  equation  20.  the 
decision  maker  would  tolerate  recommending  an  unnecessary  biopsy  for  any  prognostic 
probability  above  ftreat  all  =  1  ~  0*  =  50%. 


Appendix  2 

A  regret  theory  approach  to  decision  curve  analysis 

Iztok  Hozo1,  Athanasios  Tsalatsanis2,Andrew  Vickers3,  Benjamin  Djulbegovic24 

Indiana  University,2University  of  South  Florida,  Center  for  Evidence-based  Medicine,  3Memorial  Sloan 

Kettering  Cancer  Center,  NY  and  H.  4Lee  Moffitt  Cancer  Center  &  Research  Institute,  Tampa,  FL 

Purpose:  Decision  curve  analysis  (DCA)  has  been  proposed  as  an  alternative  method  for  evaluation  of 
diagnostic  tests,  prediction  models,  and  molecular  markers  [MDM  2006;26:565].  We  re-formulated  DCA  from 
the  regret  theory  point  of  view  to  take  into  consideration  decision  consequences  [MDM  2008;28:540]. 

Methods:  First,  we  constructed  a  classic  decision  tree  describing  three  decision  alternatives:  treat,  do  not  treat, 
and  treat/no  treat  based  on  a  predictive  model.  We  then  computed  the  expected  regret  for  each  of  these 
alternatives  as  the  difference  between  the  utility  of  the  action  taken  and  the  utility  of  the  action  that  should 
have  been  taken,  in  retrospect.  We  evaluated  the  expected  regret(s)  for  the  tree  using  different  weightings 
regarding  regret  associated  with  "omissions"  (e.g.  failure  to  treat)  vs.  "commissions"  (e.g.  treating 
unnecessary).  For  any  pair  of  strategies,  we  measure  the  difference  in  net  expected  regret  (NERD),  by 
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subtracting  the  expected  regret  of  each  alternative  from  the  other.  Finally,  using  the  concept  of  acceptable 
regret-the  range  at  which  potential  regret  associated  with  wrong  decisions  becomes  acceptable-  we  identified 
the  circumstances  where  acting  on  a  given  strategy  will  be  acceptable  even  if  the  decision  was  wrong. 

Results:  We  first  showed  that  NERD  is  equivalent  to  net  benefits  as  described  in  the  original  DCA. 

The  regret  re-formulation  of  the  original  DCA  model  showed  an  asymmetry  in  decision-making.  That 
is,  the  decision-maker  seems  to  weigh  (=the  threshold  probability  at  which  a  decision-maker  is 
indifferent  between  two  actions)  regret  associated  with  failure  to  treat  much  higher  than  the  regret 
related  to  unnecessary  treatment.  This  is  because  the  decision-maker  weights  true  positive  results  to  a 
much  greater  extent  than  false-positive  results.  Similarly,  different  attitudes  toward  omissions  vs. 
commissions  identified  different  circumstances  when  the  decision-maker  can  "live  with"  regret  even 
if  the  decision  was  wrong,  in  retrospect.  The  symmetry  in  the  decision-making  was  re-instated  when 
the  weighting  for  false-positive  and  false-negative  results  was  identical. 

Conclusions:  We  present  an  alternative  derivation  of  the  DCA  based  on  regret  theory.  Under  assumptions  of 
unequal  weighting,  the  regret  approach  generates  identical  results  to  the  original  DCA.  The  regret  approach 
may  also  be  intuitively  more  appealing  to  a  decision-maker,  particularly  in  those  clinical  situations  when  the 
best  management  option  is  the  one  associated  with  the  least  amount  of  regret  (e.g.  treatment  of  advanced 
cancer,  etc). 
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Appendix  3 


Study  ID  # 

Hospice  ID  # 

Development  of  EBM-CDSS  (Evidence-Based  Clinical  Decision  Support  System)  to  aid  prognostication  in 

terminally  ill  patients 


Demographics 

Date  of  Birth 

Date  of  Death 

Gender  (please  check  one) 

Male 

Female 

Ethnicity  (please  check  one) 

White 

African  American 

Hispanic 

Asian 

Pacific  islander 

Other: 

Date  of  Hospice  admission 

Referring  institution 

Primary  diagnosis 

Date  of  diagnosis 

Physician's  prediction  of  survival 
(Please  check  one) 

Hours-days 

Days-weeks 

<  6  months 

<  12  months  (Private  Payl) 

Patient  directives  (Please  check  one) 

Do  Not  Resuscitate  (DNR) 

Full  code 

Day  in  the  hospital  when  qualified  for 
study  (please  circle  one) 

123456789  10 

Number  of  days  in  the  hospital 

Length  of  follow  up 

Days 

Vital  Signs 

Date  Collected 

Mean  Arterial  Blood  Pressure 

mmHg 

Body  temperature 

°F 

Heart  rate 

Beats/minute 

Respiratory  Rate 

Breaths/minute 

Laboratory  Data 

Date  collected 

Serum  Creatinine 

mg/dl 

Serum  Bilirubin 

mg/dl 

Serum  Sodium 

mEq/lt 

Serum  Albumin 

g/dl 

Blood  glucose 

mg/dl 

Hematocrit 

% 

Blood  pH 

24  Hour  Urine  Output 

Cc/day 

White  blood  cell  count  (WBC) 

Cu/mm 

Alveolo-arterial  p02  gradient  (A-aD02) 

Blood  urea  nitrogen  (BUN) 

mg/dl 

Partial  pressure  carbon  dioxide  in  arterial  blood  (PaC02) 

mmHg 

Partial  pressure  oxygen  in  arterial  blood  (Pa02) 

mmHg 

Fraction  of  inspired  oxygen  in  a  gas  (FI02) 

Coronary  Artery  Bypass  Graft  (CABG)  information  (if  applicable) 

Emergency  surgery 

Yes 

No 

Ejection  fraction 

Diabetes  (medication  dependent) 

Yes 

No 

CABG  was  redone 

Yes 

No 

Number  of  grafts 

IMA  used 

Yes 

No 

MIDUR 

Yes 

No 

Co-morbidities 

Acute  Renal  Failure 

Yes 

No 

Acquired  immunodeficiency  syndrome 

Yes 

No 

Cancer 

Yes  No  Metastasis 

Chronic  obstructive  pulmonary  disease 

Yes 

No 

Congestive  heart  failure 

Yes 

No 

Cirrhosis 

Yes 

No 

Colon  cancer 

Yes 

No 

Coma 

Yes 

No 

Glasgow  coma  scale: 

Unable  to  obtain  Glasgow  coma  scale  (please  note  reason): 

Hepatic  failure 

Yes 

No 

Immuno-  suppression 

Yes 

No 

Lung  cancer 

Yes 

No 

Lymphoma 

Yes 

No 

Leukemia 

Yes 

No 

Multiple  myeloma 

Yes 

No 

Multiple  organ  system  failure  with  cancer 

Yes 

No 

Multiple  organ  system  failure  with  sepsis 

Yes 

No 

Other  co-morbidities: 
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Palliative  Performance  Scale  (Circle  One) 

PPS 

Level 

Ambulation 

Activity  &  Evidence  of  Disease 

Self-care 

Intake 

Conscious  Level 

100% 

Full 

Normal  activity  &  work  No 
evidence  of  disease 

Full 

Normal 

Full 

90% 

Full 

Normal  activity  &  work  Some 
evidence  of  disease 

Full 

Normal 

Full 

80% 

Full 

Normal  activity  with  effort  Some 
evidence  of  disease 

Full 

Normal  or 
reduced 

Full 

70% 

Reduced 

Unable  normal  job  or  work 
Significant  disease 

Full 

Normal  or 

reduced 

Full 

60% 

Reduced 

Unable  hobby/housework 
Significant  disease 

Occasional 

assistance 

necessary 

Normal  or 
reduced 

Full  or  confusion 

50% 

Mainly  sit/lie 

Unable  to  do  any  work 
Extensive  disease 

Considerable 

assistance 

required 

Normal  or 

reduced 

Full  or  confusion 

40% 

Mainly  in  bed 

Unable  to  do  most  activity 
Extensive  disease 

Mainly 

assistance 

Normal  or 

reduced 

Full  or  drowsy 
+/-  confusion 

30% 

Totally  bed 
bound 

Unable  to  do  any  activity 
Extensive  disease 

Total  care 

Normal  or 

reduced 

Full  or  drowsy 
+/-  confusion 

20% 

Totally  bed 
bound 

Unable  to  do  any  activity 
Extensive  disease 

Total  care 

Minimal  to  sips 

Full  or  drowsy 
+/-  confusion 

10% 

Totally  bed 
bound 

Unable  to  do  any  activity 
Extensive  disease 

Total  care 

Mouth  care  only 

Drowsy  or 
comma  +/- 
confusion 

0% 

Death 

Karnofsky  Performance  Scale  (Circle  One) 

100 

Normal  no  complaints;  no  evidence  of  disease 

90 

Able  to  carry  on  normal  activity;  minor  signs  or  symptoms  of  disease 

80 

Normal  activity  with  efforts;  some  signs  or  symptoms  of  disease 

70 

Cares  for  self;  unable  to  carry  on  normal  activity  or  to  do  active  work 

60 

Requires  occasional  assistance,  but  is  able  to  care  for  most  personal  needs 

50 

Requires  considerable  assistance  and  frequent  medical  care 

40 

Disabled;  requires  special  care  and  assistance 

30 

Severely  disabled;  hospital  admission  is  indicated  although  death  not  imminent 

20 

Very  sick;  hospital  admission  necessary;  active  supportive  treatment  necessary 

10 

Moribund;  fatal  processes  progressing  rapidly 

0 

Dead 

Activities  of  Daily  Living  (ADL)  Index 

For  each  area  of  functioning  listed  below,  check  the  description  that  applies.  (The  word  "assistance"  means  supervision, 

direction,  or  personal  assistance.) 

BATHING:  Sponge  bath,  tub  bath,  or  shower. 

□  Receives  no  assistance  (gets  into 

and  out  of  tub  is  the  usual  means 
of  bathing) 

□  Receives  assistance  in  bathing  only 
one  part  of  the  body  (such  as  the 
back  of  a  leg). 

0  Receives  assistance  in  bathing 

more  than  one  part  of  the  body  (or 
not  bathed). 

DRESSING:  Gets  clothes  from  closets  and  drawers,  including  underclothes  and  outer  garments  and  uses  fasteners,  including 
suspenders  if  worn. 

□  Gets  clothes  and  gets  completely 

dressed  without  assistance. 

□  Gets  clothes  and  gets  dressed 

without  assistance  except  for  tying 

shoes. 

0  Receives  assistance  in  getting 
clothes  or  in  getting  dressed,  or 
stays  partly  or  completely 

undressed. 

|  TOILETING:  Goes  to  the  room  termed  "toilet"  for  bowel  movement/urination,  cleans  self  afterward  and  arrange  clothes.  j 

□  Goes  to  toilet  room.  Clean  self,  and 
arrange  clothes  without  assistance. 
(May  use  object  for  support  such  as 
cane,  or  wheelchair  and  may 
manage  night  bedpan  or 
commode,  emptying  it  in  morning.) 

0  Receives  assistance  in  going  to 
toilet  room  or  in  cleaning  self  or 
arranging  clothes  after  elimination 
or  in  use  of  night  bedpan  or 

commode. 

0  Doesn't  go  to  toilet  room  for 
the  elimination  process. 

|  TRANSFER:  j 

□  Moves  into  and  out  of  bed  as  well 

as  into  and  out  of  chair  without 

assistance.  (May  use  object  such  as 
cane  or  walker  for  support). 

0  Moves  into  or  out  of  bed  or  chair 

without  assistance. 

0  Doesn't  get  out  of  bed. 

|  CONTINENCE:  j 

□  Controls  urination  and  bowel 

movement  completely  by  self. 

0  Has  occasional  accidents. 

0  Supervision  helps  keep  control  of 
urination  or  bowel  movement,  or 

catheter  is  used,  or  is  incontinent. 

|  FEEDING:  | 

□  Feeds  self  without  assistance. 

□  Feeds  self,  except  for  assistance  1 
cutting  meat  or  buttering  bread. 

0  Receives  assistance  in  feeding  or  is 
fed  partly  or  completely  through 
tubes  or  by  IV  fluids. 

INDEX 

A:  Independent  in  all  six  functions. 

B:  Independent  in  all  but  one  of  these  functions. 

C:  Independent  in  all  but  bathing  and  one  additional  function. 
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D:  Independent  I  all  but  bathing  dressing  and  one  additional  function. 

E:  Independent  in  all  but  bathing,  dressing,  toileting,  and  one  additional  function. 

F:  Independent  I  all  but  bathing,  dressing,  toileting,  transferring,  and  one  additional  function. 
G:  Dependant  in  all  six  functions. 

Other:  Dependant  in  at  least  two  functions,  but  not  classifiable  as  C,  D,  E,  or  F/ 

□  Indicates  Independence 

O  Indicates  Dependence 


ECOG  Performance  Status  (ECOG-PS)  (Circle  One) 

Grade 

Description 

0 

Fully  active,  able  to  carry  on  all  pre-disease  performance  without  restriction 

1 

Restricted  in  physically  strenuous  activity,  but  ambulatory  and  able  to  carry  out  work  of  a  light 
sedentary  nature,  e.g.,  light  house  work,  office 

2 

Ambulatory  and  capable  of  all  self-care  but  unable  to  carry  out  any  work  activity.  Up  and  about  more 
than  50%  of  waking  hours 

3 

Capable  of  only  limited  self-care,  confined  to  bed  or  chair  more  than  50%  of  waking  hours 

4 

Completely  disabled.  Cannot  carry  on  any  self-care.  Totally  confined  to  bed  or  chair 

5 

Dead 

Notes: 
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